Method and apparatus for temperature-gradient aware data-placement for 3D stacked DRAMs

ABSTRACT

A system including a stack of two or more layers of volatile memory, such as layers of a 3D stacked DRAM memory, places data in the stack based on a temperature or a refresh rate. When a threshold is exceeded, data are moved from a first region to a second region in the stack, the second region having one or both of a second temperature lower than a first temperature of the first region or a second refresh rate lower than a first refresh rate of the first region.

This invention was made with Government support under PathForwardProject with Lawrence Livermore National Security (Prime Contract No.DE-AC52-07NA27344, Subcontract No. B620717) awarded by DOE. TheGovernment has certain rights in this invention.

BACKGROUND

To maintain data integrity memory in dynamic random access memory(DRAM), DRAM cells are refreshed periodically. A retention time of DRAMcells is a function of operating temperature and random manufacturingvariations. As temperature increases, the DRAM cells discharge morerapidly thereby reducing the amount of time data are retained in thosecells. As a result, to reliably maintain data, thermally hotter DRAMcells need to be refreshed more often than colder DRAM cells. While therequired refresh time of a DRAM cell is partly a function oftemperature, manufacturing variations exist between each DRAM cell andthereby each subarray, bank, and vault in a DRAM die also contribute tothe refresh time needed for a particular DRAM cell. DRAM cell variationcan be both purely random (e.g., every DRAM cell is somewhat different)and parametric (e.g., DRAM cells near to one another perform similarly).Accordingly, even at similar temperatures, different storage locationsin a DRAM need to be refreshed at different rates. However, according toconventional schemes, entire regions of DRAM are refreshed at a samerate.

DRAM cells of a row cannot be accessed while the row is being refreshed,so time spent refreshing DRAM cells can reduce overall performance ofmemory-accessing workloads. High DRAM cell density combined with DRAMcell retention-time variability results in DRAM refresh becoming asevere performance bottleneck under certain conditions. Complicating thesituation, die or other layers of DRAM cells often are stacked togetherdirectly on top of each other to form a stacked DRAM, which in turnoften is stacked on top of heat-generating logical components such ascentral processing units (CPUs) and graphics processing units (GPUs),thereby increasing refresh rates when decreased refresh rates aredesired.

Conventional data placement within available DRAM cells is agnostic tovariations in DRAM retention times thereby resulting in a sub-optimaldata placement when considering certain operational characteristics ofstacked DRAM and the character of the data therein. Different sectionsand different layers of stacked DRAM are exposed to differenttemperatures depending on a variety of factors including the particulargeometry of the components and their respective heat-producingworkloads, locations of heatsinks, amount of contact with othercomponents, and the physical design of the DRAM layers and componentstherein.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure may be better understood, and its numerousfeatures and advantages made apparent to those skilled in the art byreferencing the accompanying drawings. The use of the same referencesymbols in different drawings indicates similar or identical items.

FIG. 1 is a block diagram of a stacked memory device in accordance withsome embodiments.

FIG. 2 is a block diagram of a memory structure in accordance with someembodiments.

FIG. 3 is a diagram of an example of two sets of temperature dataplotted against position of memory layers of two stacked memory devices.

FIG. 4 is a flow diagram illustrating a method for managing data in astacked memory device in accordance with some embodiments.

DETAILED DESCRIPTION

Memory bandwidth in stacked DRAM or similar stacked volatile memory is asubstantial factor in many product designs, from high-bandwidth graphicprocessor units (GPUs) to server central processor units (CPUs).Applications for machine learning, high-performance computing,cryptocurrency mining, and related high-growth domains significantlybenefit from high memory bandwidth availability in computing devices andsystems. Unfortunately, conventional refresh frequencies are fixed orleft to a memory cell refresh controller that has a limited ability tocontrol refresh rates. DRAM cells cannot be accessed during a refreshcycle. Thus, conventional approaches to stacked DRAM refresh mechanismsimpair memory bandwidth.

Rather than using a single refresh rate and seemingly randomly placingdata into certain locations or regions within one or more layers of thestacked DRAM, the techniques and systems described herein selectivelyplace and maintain data in less-frequently-refreshed areas of memory.Selective placement is especially useful for data that are frequentlyaccessed: certain data that are accessed at a substantively higherfrequency than other data. Such data are referred to as high access ratedata or data having a frequency of access that is relatively highcompared to other data. In some embodiments, variable refresh rates areused within stacked DRAM regions. A stacked DRAM die may bepre-characterized to identify certain memory ranges that need to berefreshed at a first rate (e.g., once per X milliseconds (ms)) at aparticular temperature, while other ranges only need to be refreshed ata lower or second rate (e.g., once per Y ms) at the same temperature.The regions having a lower refresh rate are preferred regions forplacing data that are frequently accessed. Data are thereby placed intocertain stacked DRAM memory regions having lower refresh rates thanother regions.

A memory refresh system as described herein monitors localizedtemperatures, and thereby localized refresh rates, within the stackedDRAM to understand and take advantage of a respective actual or requiredrefresh rate of each region of memory. According to certain embodiments,DRAM retention time variations are exposed to a hardware component(e.g., a memory cache controller) or to a system software component(e.g., an operating system (OS) or a hypervisor). The hardware orsoftware component performs a retention-aware data placement therebyimproving memory access performance and reducing the chance for memoryaccess collisions. Under this approach, refresh rate changes aredetected, and data are moved to a new location based on the detectedrefresh rate changes.

FIG. 1 illustrates a stacked memory device 100 in accordance with someembodiments. The stacked memory device 100 includes a processor 101 onwhich a plurality of memory layers, such as the illustrated three memorylayers 102-104, are stacked. Vias 110 and traces in and between thelayers allow the various structures to address and to communicate witheach other across the memory layers 102-104. The various layers aredepicted as being physically separated from each other for sake ofsimplicity and to permit illustration of various components therein. Theprocessor 101 includes one or more processor cores 106 for executingcomputing instructions or micro operations. While a single processorcore 106 is illustrated, the processor 101 may instead implementmultiple processor cores, including processor cores of different types.A memory controller 107 manages the various sections of the memorylayers 102-104. Alternatively, each layer includes its own memorycontroller or memory sub-controller and operation of the respectivememory sub-controllers is coordinated. Each memory layer 102-104implements a DRAM architecture and includes a plurality of DRAM cells,each DRAM cell storing a bit of data.

The DRAM cells are divided into a plurality of regions, such as ranks,banks, arrays, sub-arrays, and the like. For example, the first memorylayer 102 includes a first region 111 and a second region 112. Thesecond memory layer 103 includes a first region 113 and a second region114. The third memory layer 104 includes four regions: a first region115, a second region 116, a third region 117, and a fourth region 118.According to some embodiments, each region 111-118 includes at least onethermal sensor 125 and at least one memory refresh module 126. Accordingto certain embodiments, the memory controller 107 includes a pluralityof vault controllers, each managing a sub-stack of memory regions withinthe stacked memory device 100. For example, the stacked memory device100 may include a vault controller (not illustrated) which is configuredto control a set of first regions 111, 113, 115 and data stored therein.For example, a first vault controller is configured to maintain datawithin the set of first regions 111, 113, 115 as further describedherein.

In operation, each of the regions 111-118 or each of the memory layers102-104 in the stacked memory device 100 is determined to be at aparticular temperature T_(N) 105 such as an average temperature, where“N” refers to the particular region 111-118 or particular memory layer102-104. For example, the processor 101 is at a processor temperatureT₀, the first memory layer 102 is at a first temperature T₁, the secondmemory layer 103 is at a second temperature T₂, and the third memorylayer 104 is at a third temperature T₃. Alternatively, the first region111 of the first memory layer 102 is at a first temperature T₁ at afirst time or during a first interval, the first region 113 of thesecond memory layer 103 is at a second temperature T₂ at a first time orduring a first interval, the first region 115 of the third memory layer104 is at a third temperature T₃ at a first time or during a firstinterval, and so forth, layer by layer and region by region according toa density and availability of thermal sensors 125. Often, thetemperatures from the processor 101 upward through the memory layers102-104—T₀ through T₃—are individual temperatures along a temperaturegradient from a high temperature to relatively low temperature as thememory device 100 releases energy in the form of heat to itssurroundings. The temperatures of the particular regions within onelayer can be substantially the same as one another, but such a situationis not always so. As understood by those in the art, temperatures atvarious points in the stacked memory device 100, when operating,typically vary along each of three dimensional axes 130 depending on thevarious heat-generating activities of the components in the memorydevice 100.

The memory controller 107 refreshes each of the DRAM cells, rank byrank, bank by bank, or region by region 111-118 via various memoryrefresh modules 126 based on a respective temperature T_(N) 105 measuredfor the various regions 111-118 by the respective thermal sensors 125. Ahigher temperature is correlated with a faster refresh cycle needed toreliably maintain data in the various data bits in the regions 111-118.That is, according to some implementations, the memory refresh modulesuse a correlation between temperature and memory refresh rate whenoperating. According to certain embodiments, a refresh rate is set foreach of the various regions 111-118 based on a respective temperature ofeach of the respective regions 111-118. Thus, the refresh rate for agiven region 111-118 is periodically or otherwise dynamically updated asthe various temperatures of these regions 111-118 change over time.

Not only are certain data initially placed in a preferred location basedon temperature or refresh rate, data are moved to a different regionduring operation of the stacked memory device 100. The following is anexample of moving data from one memory region to another. In the stackedmemory device 100, first data 121 are stored in a first memory bank 123of a second region 112 of the first memory layer 102. For purposes ofexample, the first memory bank 123 and second region 112 are determinedto be at a first temperature T₁ at a particular time t. At this time,the first temperature T₁ is higher than a second temperature T₂, and thesecond temperature T₂ is higher than a third temperature T₃. Forpurposes of this example, the third temperature T₃ is a preferredtemperature and preferred memory refresh rate in the stacked memorydevice 100. For sake of illustration of the techniques described herein,the third temperature T₃ is a lowest temperature in the stacked memorydevice 100 and corresponds to a preferred region to receive data movedfrom a hotter region. The first data 121 are moved to a second memorybank 124 of the third region 117 of the third memory layer 104.

In this example, a position of the second memory bank 124 is in adifferent location or region with respect to a first directional axis, asecond directional axis, and a third directional axis in the stackedmemory device 100. In certain embodiments, the first data 121 are movedalong one, two, or three directional axes, depending on a configurationor logical programming of the memory controller 107. That is, in certaincircumstances, it is required or beneficial to persist data in a samelayer or region within the stacked memory device 100 but to a place oflower temperature and thereby lower memory cell refresh rate. Accordingto certain embodiments of steps of operation, the first data 121 arefirst identified as highly accessed data prior to moving the first data121 to a new region and a new memory bank that is cooler in temperature.That is, the first data 121 are moved when the first data 121 experienceor begin to experience a high rate of access such as access by anoperating system (OS) or computer program. According to certainembodiments, for faster operation, a per-layer data structure ismaintained by the memory controller 107 for identifying locations orregions within the layers. Alternatively, a per-layer data structure ismaintained as a separate structure in each of the memory layers 102-104which includes locations for each of a set of available, unoccupied, orfree memory pages or regions to which data may be moved. When triggeredto move data, the per-layer data structure is accessed and a destinationlocation is identified based on the references to free memory pages orregions therein.

In certain circumstances, movement to an unoccupied memory region is notpossible, such as when the memory regions 111-118 are substantiallyfilled with data. In such circumstances, the first data 121 are switchedwith second data 122 which are preferably data that are accessed lessfrequently or least frequently accessed as determined for each group ofdata presently stored in the stacked memory device 100. Generally, ahardware or software component operates to place certain data such asfrequently-accessed data and data of certain cachelines into one or morememory regions that are refreshed at a relatively low rate, whichcorresponds to one or more memory regions that have a high dataretention time. According to certain embodiments, in a system where astacked memory device 100 is part of a main memory, an OS migrates themost frequently accessed data to memory regions with longer retentiontimes. By placing the most frequently accessed data entities, eitherhardware-managed cachelines or software-managed memory pages, in amemory region with better retention times, the loads/stores issued bythe processor 101 are interrupted less often by DRAM refresh commands ofthe memory controller 107. Moving at least some of the data to a newlocation or region results in increased memory bandwidth among improvedperformance metrics of the system.

FIG. 1 also illustrates moving data from one region to another regionwithin a same memory layer in a stack of memory layers. In operation,and at certain times, each of the various sub-divisions of the memorylayers 102-104 are at respective temperatures as measured by the thermalsensors 125. For example, a third memory bank 133 of the fourth region118 is at a fourth temperature T₄ while a fourth memory bank 134 of thesame fourth region 118 is at a fifth temperature T₅. In this example,the fourth temperature T₄ is higher than the fifth temperature T₅ andthe values in the DRAM cells of the third memory bank 133 are refreshedat a higher rate (i.e., more frequently) than the values in the DRAMcells of the fourth memory bank 134 as determined by the memorycontroller 107.

At a certain point in time, the memory controller 107 is triggered tomove third data 132 in the third memory bank 133 into the fourth memorybank 134 to take advantage of the slower refresh rate in the fourthmemory bank 134 at the lower fifth temperature T₅. Such triggering ofthe memory controller 107 is performed by an operating system, by memorycontroller logic of the memory controller 107, or by another component.For example, in response to the third data 132 being determined to befrequently accessed data, the memory controller 107 identifies a set offree or available DRAM cells in the fourth memory bank or identifies aset of DRAM cells that could be swapped from the fourth memory bank 134to the third memory bank 133 in the event that the fourth memory bank134 is full. During an appropriate time period (e.g., when the processor101 is in an idle state, a low activity state, or a low-power state),the memory controller 107 moves the third data 132 to the fourth memorybank 134, and, if making a swap, moves the fourth data 131 to the thirdmemory bank 133. The memory controller 107 does so by copying the thirddata 132 and the fourth data 131 row by row, or region by region, asunderstood by those in the art. In this example, data are relocatedalong one of three dimensional axes 130 from a region of hightemperature T₄ to a region of low temperature T₅.

FIG. 2 is a block diagram of a memory structure 200 as part of acomputing system in accordance with some embodiments. The memorystructure 200 is a two-dimensional representation of a stacked memorystructure having multiple memory layers, such as the memory layers102-104 of FIG. 1. The memory structure 200 includes a memory 201 havinga memory controller 220 that manages memory layers, memory regions, andDRAM cells in the memory 201 including those of a set of stacked memorylayers. The memory 201 also includes an operating system 210 storedtherein, such as during runtime of the computing system. For sake ofsimplicity, memory regions, individual DRAM cells, and stacked memorylayers are not shown.

The memory controller 220 includes one or more refresh modules 221. Therefresh module 221 includes and operates refresh logic 223 to cycle andmaintain data in the DRAM cells, such as the first data 121 of a firstmemory layer 102 of a stacked memory device 100 of FIG. 1. The refreshmodule 221 and the refresh logic 223 acquire and use refresh data 222 toschedule refreshing of the various memory banks and memory regions.According to certain embodiments and as described in greater detailbelow, the refresh data 222 include data representing a frequency ofaccess of certain data stored in the various layers of stacked memoryand a current refresh schedule or rate of refresh for the data managedby the memory controller 220. The current refresh rate may be a minimumrequired refresh rate or another refresh rate. The refresh data 222 arebased on various measurement data and other data available in datacontrol structures 202 about the various memory regions including thosememory regions of stacked memory layers.

The data control structures 202 include a set of data for each of thelayers of a stacked memory such as the layers 102-104 of FIG. 1. Forexample, a first set of data 203 corresponds to a first memory layersuch as the first memory layer 102, and a Nth set of data 206corresponds to an Nth or last memory layer, such as the third memorylayer 104 of FIG. 1. Each set of data of the data control structures 202includes, for example, a set of measurements 204, 207, which include,for example, a temperature from respective thermal sensors of andassociated with various memory regions. As an example, the firstmeasurements 204 illustrated in the first set of data 203 aretemperatures of locations 205 in a first layer of a stacked memory, andthe second measurements 207 illustrated in the Nth set of data 206 aretemperatures of locations 208 in an Nth layer of a stacked memory. Inother embodiments, measurement data include values for one or acombination of voltages, currents, refresh rates, or the like taken fromthe various locations 205, 208 in the memory layers. While location isdescribed with respect to the data control structures 202, location canrefer to or be associated with a region within the memory structure.

The measurement data are correlated with various available and actualrefresh rates or schedules for maintaining data in DRAM cells of memorylayers, such as layers of stacked DRAM. The refresh module 221 accessesthe data of the sets of data 203, 206 about the memory layers bycommunicating with the data control structures 202. According to certainembodiments, the data of the sets of data 203, 206 about the layers arestored in a central location. In other embodiments, the data of the setsof data 203, 206 are stored in individual structures in each layer ofstacked memory.

In operation, the memory controller 220 implements movement of data fromone region to another region, such as moving user data or operatingsystem data (e.g., the first data 121, the second data 122), accordingto certain embodiments. In other embodiments, the operating system 210performs the movement by directing the memory controller 220 to make thedata movement. According to some embodiments, the operating system 210includes a memory management module 211. The memory management module211 includes, or has access to, memory management data 212 obtainedfrom, or derived from, the data of the data control structures 202. Forexample, the memory management data 212 are derived from the variousstacked memory layers from the measurements 204, 207 and the locationdata 205, 208 of the various memory layers 1-N. Memory management logic213 of the memory management module 211 keeps track of, for example,data access frequencies and temperatures or refresh rates at the variouslocations throughout the stacked memory layers.

When the memory management logic 213 decides to move data, the memorymanagement module 211 communicates with the memory controller 220 tomove certain data to a new location based on a new temperature, a newrefresh rate, or a new access frequency. For example, when a region ofmemory includes frequently accessed data, and the particular regionexceeds a threshold temperature thereby raising a memory refresh rate,the memory controller 220 is programmed to move data identified ashighly accessed to a new location such as at a location with a lowesttemperature. According to at least some embodiments, the thresholdtemperature is a pre-determined value. Alternatively, movement of datais performed when there is a detected or determined change over time toone or more of the following variables: temperature of a memory region,a refresh rate of the memory region, and an access frequency of thememory region.

The following example is illustrative of the operation of the memorystructure and data placement. Highly accessed data remain highlyaccessed by one or more processes of the operating system 210. Thememory management logic 213 thus determines that an improvement to anoverall memory bandwidth usage can be obtained by moving first data to anew memory region within a stacked memory that is a lower temperatureregion in a same or in a different memory layer of a plurality ofstacked memory layers. At a time determined by the memory managementlogic 213, the memory management module 211 communicates with the memorycontroller 220, which in turn effectuates the data movement. As needed,the data movement involves two data movements: making room in a lowertemperature memory region by first moving second data from the lowertemperature memory region and replacing the second data with first andhighly-accessed data. The second data are placed in the first locationwhere the first and highly-accessed data were previously located, oralternatively may be moved to a third location. By swapping the twobatches of data, the operating system lowers the frequency at whichfirst and highly-accessed data are refreshed in stacked DRAM cellsbecause the new location is in an area of the stacked memory that is ata lower temperature. In summary, data may be moved and managed within amemory structure by one or more hardware components, by one or morecomponents of an OS, or a combination of hardware components andcomponents of the OS.

FIG. 3 illustrates an example chart 300 of two sets of temperature dataplotted against position of memory layers of two stacked DRAM memorydevices. The chart 300 includes a first set of temperature data points301 for a first stacked memory device having eight memory layers stackedon top of one another. A first line 302 is fitted to the first set oftemperature data points 301. The chart 300 also includes a second set oftemperature data points 303 for a second stacked memory device havingtwelve memory layers stacked on top of one another. A second line 304 isfitted to the second set of temperature data points 303. The x-axis ismarked with layer position for the data points 301, 303 from a base ofthe respective first and second devices to their respective tops orlids. The y-axis is marked with a maximally observed temperature of eachlayer in degrees Celsius (deg. C.). A maximum difference 305 ofapproximately eight deg. C. is observed between a temperature of therespective first memory layers of the first device and the seconddevice. On average, there is a temperature difference of approximately1.65 deg. C. for the first eight-layer device and a temperaturedifference of approximately 1.55 deg. C. for the second twelve-layerdevice. With each of the first device and second device, a plurality ofDRAM memory refresh rates are correlated to a respective temperaturethreshold or temperature range.

FIG. 4 illustrates a method 400 for managing certain data units in astacked memory device in accordance with some embodiments. For example,the memory device is a stacked DRAM device such as the stacked memorydevice 100 of FIG. 1. The data units managed by the method 400 belong toa list or other identified group, such as certain data units that arefrequently accessed by one or more processes of an OS. The processes ofthe method 400 are repeated for each data unit being managed. Further,the method 400 is based on an assumption that for each iteration of themethod 400, each managed data unit may remain in a certain location ormay be moved to an improved location in the stacked memory devicedepending on currently existing conditions in the particular stackedmemory device.

For purposes of illustration, a data unit is identified as a page,pointed to by a page table of the OS, and its underlying physicaladdress, physical address range, or physical addresses when describingthe method 400. However, in other embodiments, the data unit may bebigger than, or smaller than, a page. A page can be stored in one ormore locations such as a memory bank, memory rank, or memory layer in amulti-layer stacked memory device.

At block 401, an OS or memory controller (e.g., memory controller 107,FIG. 1) determines an access count for the data unit since a lastrefresh cycle. According to some embodiments, the access count is acurrent access count. According to other embodiments, an access countcorresponds to a number of data refreshes or cycles since a last accessto a particular data page.

At block 402, the OS or the memory controller obtains a new layer forthe data page based on an access count and based on a current layer oran old layer of the data page (i.e., layer identity). In someembodiments, if the access count or access rate (i.e., the metric as thebasis for moving the data unit) does not exceed an access count, anaccess threshold, an access threshold value, or a threshold difference,then no new layer is identified. Obtaining a new layer at block 402includes comparing, by the OS or the memory controller, the access countor access count rate per unit time of the current layer against those ofavailable layers and identifying a more suitable new layer in thestacked memory device. According to at least some embodiments, the OS orthe memory controller adjusts a threshold value over time depending on atemperature difference or other difference between metrics of therespective layers. For example, if the temperature difference betweenlayers narrows from 2.5 deg. C. at a first time to 2.1 deg. C. at asecond time, the threshold value or threshold difference for triggeringmovement of data is increased so that a decision to move data occursless often because the benefit of moving the data is decreased. Thischange to the threshold value compensates for a reduced benefit ofmoving data to a cooler layer when a temperature benefit decreases and atemperature gradient narrows, and thereby compensates for the memoryrefresh cycle rate benefit decreasing from the first time to the secondtime. In this example, when a refresh rate difference at 2.5 deg. C. is10%, and the refresh rate difference at 2.1 deg. C. is 6%, then raisingthe threshold value before moving data to the new layer compensates forthe decrease in the memory cell refresh rate at the second time. Inoperation, the threshold values, refresh rates, and so forth may bedynamically adjusted, determined, or calculated such as by the OS or thememory controller, or may be fixed pre-determined values consistent withdesign parameters and initial configurations.

Generally, the new layer is cooler in temperature than the current layercorresponding to a lower rate of memory cell data refreshing than thecurrent layer. Therefore, according to certain embodiments, moving datato a new layer by the OS or the memory controller also includesidentifying refresh rates or temperatures such as through samplingrefresh rates over time or sampling temperatures over time of variousregions in the stacked memory to keep the OS, the memory controller, orboth the OS and the memory controller apprised of available locations(e.g., available layers, available memory cells) to which to move data.Depending on the particular components of the regions and availablemetrics therefrom, a temperature may be obtained through a calculationbased on a correlation between temperature memory refresh rate, and viceversa. Such can be done by sampling a voltage or a current value of acomponent, and correlating the sampled voltage or current value witheither a temperature or a memory cell refresh rate. The sampling can beperformed by either the OS, the memory controller, or some othercomponent. Correlating includes selecting a value from a data structurehaving a plurality of values. Correlating also includes calculating avalue based on a formula or equation using values available in thesystem, stored in a data structure in memory, or combination thereof.

At block 403, the OS or the memory controller compares the newlyobtained or identified layer and the current layer. If the new layer isthe same as the current layer for the data unit, the cycle method 400does nothing for the particular data unit. The method 400 is repeatedfor each of the managed data units. Once complete for a particularcycle, the method 400 includes waiting for a top of a next cycle as themethod 400 is in operation as long as certain data units are managed bythe OS, the memory controller, or a combination of software and hardwarecomponents.

When there is a newly identified destination layer for a particular dataunit, the method 400 continues. At block 404, the OS or the memorycontroller determines whether there is a new page available in the newlyidentified layer. If so, at block 409, the data of the data unit arephysically copied, by the OS or the memory controller, from the locationor locations in the current memory layer to the new location orlocations in the new layer, and the entry in the page table is updatedto reflect the new physical location or locations in the new layer.

When there is not a new page available in the newly identified layer,the method 400 continues to block 405, whereupon the OS or the memorycontroller determines one of at least two alternative actions that canbe performed. At block 405, the OS or the memory controller determineswhether the system (e.g., hardware, firmware, boot-sequence, OS) isconfigured for swaps between layers or between the two particular layersat issue. For example, in the case of an OS variation, the OS checkswhether swaps between layers are enabled. Such an option can be set orchanged through a system control (sysctl) interface. If swaps are notenabled, at block 406, the method 400 continues by having the OS or thememory controller obtain or identify a new layer and looping back to,for example, the process at block 404 and determining whether a new pagein the newly identified layer is available. If there are no more layersto consider, or if the swaps between layers option is not enabled atblock 405, the method 400 ends. Alternatively, if the swaps betweenlayers option is not enabled at block 405 for a particular layer, themethod 400 passes from block 406 and back to block 401 for a nextiteration. For block 406, obtaining the new layer includes the same orsimilar processes as described above when first identifying a new layer.

When swaps are enabled at block 405, the method 400 continues to block407 et seq., which represent a second alternative action includingswapping out data in the new layer to make room for the data to bemoved, the data being managed by the managed data unit underconsideration. At block 407, this second alternative includes the OS orthe memory controller identifying a page and underlying data in the newlayer to swap out, and, at block 408, the OS or the memory controllerswapping the data of the newly identified page with the data of themanaged data unit destined for the new layer. By way of example, theswap is performed by triggering a swap operation performed by a memorycontroller such as the memory controller 107 of FIG. 1 or the memorycontroller 220 of FIG. 2. Once the managed data unit has been swapped atblock 408, the method 400 is repeated for the next managed data unit.

To implement the method 400, an OS, a hypervisor, a memory controller,or a combination thereof, maintains and uses certain data structures formaking comparisons and performing logic functions as known to those inthe art. According to certain embodiments, a refresh rate to maintaindata within the DRAM cells of a stacked memory device is unrelated to arate of performing the method 400 or a memory temperature optimizationrate. The method 400 is performed on a periodic basis according to anupdate frequency determined by the OS or set by configuring one or moreparameters in the OS, the hardware, the firmware, or other component ina system. Alternatively, the method 400 is performed based on certaintemperature events such as detecting a threshold change in one or morelayers or one or more locations or regions within the stacked memorydevice.

According to some embodiments, the method 400 is performed based on anoptimization scheme as follows. A new location, a new region, or a newlayer is identified by optimizing or increasing an overall memorybandwidth by solving the following mathematical formulation:

$\begin{matrix}{{\underset{A}{Maximize}{\sum\limits_{i = 0}^{N - 1}{{BW}_{i}\mspace{14mu}{subject}\mspace{14mu}{to}\mspace{14mu}{\sum\limits_{i = 0}^{N - 1}\alpha_{ij}}}}} = 1} & (1)\end{matrix}$

Moving the data of a data unit between two locations could incurperformance penalties so just moving all data to a cooler memory regionis not necessarily improving memory access in all situations and in allconditions. For example, if a first memory region or area has a dataaccess rate A₁ such that the memory bandwidth is saturated, moving allthe data from a first location to a second location does not improve thenet memory bandwidth of the stacked memory device. Instead, moving onlya portion of the data of the data unit (or one data unit instead of allmanaged data units) from the first location to the second location(e.g., memory bank, memory layer, memory region associated with aparticular refresh rate or temperature) can more optimally exploit thebandwidth of both memory locations thereby improving memory bandwidthperformance of the stacked memory device. Accordingly, optimization ofmemory bandwidth use is performed by the OS or the memory controlleraccording to the equation (1) above where N is the total number oflayers of the stacked memory device, and A={α_(ij)|i,j={0, 1, 2, . . . ,N−1}} is an N×N matrix. A memory use bandwidth (BW_(i)) per layer isgiven by:

$\begin{matrix}{{BW}_{i} = {\sum\limits_{i = 0}^{N - 1}{\alpha_{ij}n_{j}}}} & (2)\end{matrix}$

where n_(j) is a number of memory access requests going to layer jbeyond a given threshold and α_(ij) is a fraction of requests from layerj mapped to layer i. The solution is α*_(ij), an optimal distribution ofrequests that maximizes the memory bandwidth for the entire stackedmemory device.

In some embodiments, the apparatus and techniques described above areimplemented in a system including one or more integrated circuit (IC)devices (also referred to as integrated circuit packages or microchips),such as the stacked memory devices having DRAM cells described abovewith reference to FIGS. 1-4. Electronic design automation (EDA) andcomputer aided design (CAD) software tools may be used in the design andfabrication of these IC devices. These design tools typically arerepresented as one or more software programs. The one or more softwareprograms include code executable by a computer system to manipulate thecomputer system to operate on code representative of circuitry of one ormore IC devices so as to perform at least a portion of a process todesign or adapt a manufacturing system to fabricate the circuitry. Thiscode can include instructions, data, or a combination of instructionsand data. The software instructions representing a design tool orfabrication tool typically are stored in a computer readable storagemedium accessible to the computing system. Likewise, the coderepresentative of one or more phases of the design or fabrication of anIC device may be stored in and accessed from the same computer readablestorage medium or a different computer readable storage medium.

A computer readable storage medium may include any non-transitorystorage medium, or combination of non-transitory storage media,accessible by a computer system during use to provide instructionsand/or data to the computer system. Such storage media can include, butis not limited to, optical media (e.g., compact disc (CD), digitalversatile disc (DVD), Blu-Ray disc), magnetic media (e.g., floppy disc,magnetic tape, or magnetic hard drive), volatile memory (e.g., randomaccess memory (RAM) or cache), non-volatile memory (e.g., read-onlymemory (ROM) or Flash memory), or microelectromechanical systems(MEMS)-based storage media. The computer readable storage medium may beembedded in the computing system (e.g., system RAM or ROM), fixedlyattached to the computing system (e.g., a magnetic hard drive),removably attached to the computing system (e.g., an optical disc orUniversal Serial Bus (USB)-based Flash memory), or coupled to thecomputer system via a wired or wireless network (e.g., networkaccessible storage (NAS)).

In some embodiments, certain aspects of the techniques described abovemay implemented by one or more processors of a processing systemexecuting software. The software includes one or more sets of executableinstructions stored or otherwise tangibly embodied on a non-transitorycomputer readable storage medium. The software can include theinstructions and certain data that, when executed by the one or moreprocessors, manipulate the one or more processors to perform one or moreaspects of the techniques described above. The non-transitory computerreadable storage medium can include, for example, a magnetic or opticaldisk storage device, solid state storage devices such as Flash memory, acache, random access memory (RAM) or other non-volatile memory device ordevices, and the like. The executable instructions stored on thenon-transitory computer readable storage medium may be in source code,assembly language code, object code, or other instruction format that isinterpreted or otherwise executable by one or more processors.

Not all of the activities or elements described above in the generaldescription are required, that a portion of a specific activity ordevice may not be required, and that one or more further activities maybe performed, or elements included, in addition to those described.Still further, the order in which activities are listed are notnecessarily the order in which they are performed. Also, the conceptshave been described with reference to specific embodiments and DRAMcells. However, one of ordinary skill in the art appreciates thatvarious modifications and changes can be made without departing from thescope of the present disclosure as set forth in the claims belowincluding an understanding that while DRAM cells are referenced, thedisclosure is applicable to any memory cell that benefits from orrequires a refresh operation. Accordingly, the specification and figuresare to be regarded in an illustrative rather than a restrictive sense,and all such modifications are intended to be included within the scopeof the present disclosure.

Benefits, other advantages, and solutions to problems have beendescribed above with regard to specific embodiments. However, thebenefits, advantages, solutions to problems, and any feature(s) that maycause any benefit, advantage, or solution to occur or become morepronounced are not to be construed as a critical, required, or essentialfeature of any or all the claims. Moreover, the particular embodimentsdisclosed above are illustrative only, as the disclosed subject mattermay be modified and practiced in different but equivalent mannersapparent to those skilled in the art having the benefit of the teachingsherein. No limitations are intended to the details of construction ordesign herein shown, other than as described in the claims below. It istherefore evident that the particular embodiments disclosed above may bealtered or modified and all such variations are considered within thescope of the disclosed subject matter. Accordingly, the protectionsought herein is as set forth in the claims below.

What is claimed is:
 1. A method comprising: monitoring a first memoryrefresh rate of a first location in a multi-layer stacked dynamic randomaccess memory (DRAM), the first location including memory cells;responsive to a change in the first memory refresh rate exceeding afirst threshold, moving data from the first location to a secondlocation in the multi-layer stacked DRAM, the second location having asecond temperature different from a first temperature of the firstlocation; and maintaining a per-layer data structure, wherein theper-layer data structure stores a reference for free memory pages ineach layer of the multi-layer dynamic random access memory, and whereinmoving the data from the first location to the second location includes:accessing the per-layer data structure; and identifying the secondlocation based on the references of the free memory pages therein, thesecond location being in a different layer than a layer of the firstlocation.
 2. The method of claim 1, wherein monitoring the first memoryrefresh rate includes sampling a temperature of the first location inthe stacked DRAM.
 3. The method of claim 1, wherein: the first thresholdis based on a second refresh rate of a second location of the stackedDRAM.
 4. The method of claim 3, further comprising: monitoring thesecond location in the stacked DRAM for changes in the second refreshrate; storing a representation of the second refresh rate of the secondlocation in a storage location of the stacked DRAM; and determining thefirst threshold using the stored representation of the second refreshrate.
 5. The method of claim 1, wherein the first location and thesecond location of the stacked DRAM are in different hardware-managedcachelines.
 6. The method of claim 1, wherein the first location and thesecond location of the stacked DRAM are in different software-managedmemory pages.
 7. The method of claim 1, wherein monitoring the change inthe first memory refresh rate includes: identifying a current refreshrate based on a correlation between the first temperature of the firstlocation and the first memory refresh rate.
 8. The method of claim 1,further comprising: monitoring the first temperature of the firstlocation of the stacked DRAM; and responsive to the first temperatureexceeding a temperature threshold, moving the data to the secondlocation.
 9. The method of claim 1, wherein the data in the firstlocation are accessed with a higher frequency relative to accesses ofother data in the stacked DRAM.
 10. The method of claim 1, furthercomprising: monitoring a frequency of access of the data at the secondlocation by an operating system; and responsive to the frequency ofaccess exceeding a second threshold, moving data from the secondlocation to a third location in the stacked DRAM, the third locationhaving a third temperature different from the second temperature of thesecond location.
 11. An apparatus comprising: a stack of two or morelayers of volatile memory; a per-layer data structure storing areference for free memory pages in each layer of the volatile memory; amemory management module configured to: determine whether a number ofactivations of a first region of a first layer of the stack within atime period exceeds a first threshold; monitor a first temperature ofthe first region; and determine whether the first temperature exceeds asecond threshold; and a memory controller configured to: responsive tothe first threshold being exceeded, access the per-layer data structureand identify a second region in a second layer, the second layerdifferent from the first layer, based on the references of the freememory pages therein; and move data stored in the first region to thesecond region in the stack of two or more layers of volatile memory, thesecond region having a second temperature lower than the firsttemperature of the first region.
 12. The apparatus of claim 11, whereinthe memory management module is further configured to: determine anumber of activations per time at a plurality of sampled locations ofmemory cells in the stack of two or more layers of volatile memory; andidentify the second region as a sampled region having a lowest memorycell refresh rate for the time period.
 13. The apparatus of claim 12,wherein the memory management module is further configured to: sample avoltage or current value of a component in the first region; correlatethe voltage or current value with a temperature; and correlate thetemperature with a memory cell refresh rate.
 14. A system comprising: amulti-layer stacked dynamic random access memory (DRAM) device includinga first region in a first layer and a second region in a second layer,wherein the DRAM device includes a per-layer data structure, theper-layer data structure storing a reference for free memory pages inthe first layer and the second layer; a memory management module todetermine whether a first temperature of the first region of the DRAMdevice exceeds a temperature threshold at a first time based on afrequency of access of data of the first region; and a memory controllercoupled to the DRAM device and configured to control the DRAM device to:responsive to the memory management module determining the firsttemperature exceeds the temperature threshold at the first time byidentifying the first region as a high data access rate region: movedata from the first region to the second region at a second time afterthe first time based on accessing the per-layer data structure andidentifying the second region based on the reference of free memorypages therein.
 15. The system of claim 14, wherein the memory managementmodule is further configured to: sample the first temperature over timeincluding the first time; sample a second temperature of the secondregion over time including the first time; and select the temperaturethreshold from a plurality of temperature thresholds based on the firsttemperature and the second temperature at the first time.